Skip to content

fix(vigild): the reaper no longer loses a service's exit status - #1

Merged
git001 merged 1 commit into
git001:mainfrom
daveman1010221:fix/reaper-steals-service-exit
Nov 2, 2026
Merged

git001 merged 1 commit into
git001:mainfrom
daveman1010221:fix/reaper-steals-service-exit

Conversation

@daveman1010221

Copy link
Copy Markdown
Contributor

Problem

When vigild runs as PID 1 (or with --reaper), the reaper drains every exited child with waitpid(-1, WNOHANG) on each SIGCHLD. That also collects vigild's own children, services and exec checks, which tokio is waiting on. A pidfd tells tokio its child exited, but it doesn't stop another waitpid from collecting the child first. When the reaper wins that race, tokio's wait fails with ECHILD and the exit status is lost:

  • A service that exited 0 is reported as success=false exit_code=1 and handled by its on-failure policy: restarted, or failure-shutdown with exit code 10.
  • An exec check that passed counts as failed.

We hit this on a container whose one-shot service runs while other services are starting: it exited 0, and vigild reported it as failed on two of three starts.

Fix

  • The reaper now keeps each status it collects, for 60 s.
  • A new reaper::wait_child replaces Child::wait in the service actor and in the exec probe. When the wait fails with ECHILD, it takes the status the reaper kept. The reaper records a status right after its waitpid, so a waiter can see ECHILD just before the record exists; wait_child keeps looking for up to 200 ms.
  • Statuses nobody claims are dropped after the 60 s window, so the table can't grow without bound.

Without the reaper, wait_child is just Child::wait.

Tests

The new tests in reaper.rs collect a child's status with waitpid before wait_child runs, just as the reaper would. They check that each kind of status survives: an exit code, a success and a signal. They also check that the path without the reaper is unchanged, and that old statuses are dropped.

Checked locally on Linux with Rust 1.99, running the same steps as CI:

  • cargo fmt --all --check and cargo clippy --workspace --all-targets -- -D warnings are clean.
  • cargo test --workspace passes, except for do_start_passthrough_log_mode and do_start_disabled_log_mode. Those two also fail on unchanged main on this machine (NixOS), because they run /bin/sleep, which NixOS doesn't have. They should pass on CI's Ubuntu runner.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QAe3biqjeQJRMfngB95eyW

As PID 1 (or with --reaper), vigild's reaper drains every exited child
with waitpid(-1, WNOHANG) on each SIGCHLD. That collects vigild's own
children too, services and exec checks, which tokio waits on: a pidfd
tells tokio its child exited, but does not stop another waitpid from
collecting it first. When the reaper won, tokio's wait failed with
ECHILD and the exit status was lost: a service that exited 0 was
reported as `success=false exit_code=1` and handled by its on-failure
policy (restarted, or failure-shutdown with exit code 10), and an exec
check that passed counted as failed.

Seen on a container whose one-shot service runs right as other services
start: it exited 0 and was reported failed on two of three starts.

The reaper now keeps each status it collects, for 60 s, and
reaper::wait_child, which the service actor and the exec probe use in
place of Child::wait, takes it from there when the wait fails with
ECHILD.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QAe3biqjeQJRMfngB95eyW
@git001
git001 merged commit 68405c5 into git001:main Nov 2, 2026
@git001

git001 commented Nov 2, 2026

Copy link
Copy Markdown
Owner

thanks for contribution v3.2.2 will be released soon

daveman1010221 added a commit to daveman1010221/vigil-rs-nix that referenced this pull request Nov 2, 2026
… goes

The fix was merged upstream as git001/vigil-rs#1 and released in v3.2.2, so it no longer needs carrying. Tried on a five-node cluster: every node booted on it, the services came up, and a killed kube-proxy was reported as failed and restarted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A6J8LPYCZWcnSD1dESmJpX
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants